Oasis Support Engineering : OASIS Mooring turn
This page last changed on Jun 14, 2021 by kgomes.





WARNING: This page is now located in the SE-IE Markdown Git project located here. You can check out that project, edit the documentation and push back to BitBucket and it will automatically be deployed on MBARI's documenation site



This is the procedure to take when the Observatory Support Group turns an OASIS mooring.  A turn means that the currently deployed mooring is recovered and a new one is deployed with a different set of instruments at the same nominal location.

To ease the stress of SSDS data processing continuity on the day of the turn all the instrument XML and metadata processing can be tested with the dockside test deployment that is already part of sta/ndard OSG operating procedure. This test deployment will get added to SSDS, clearly labeled as a test deployment and the netCDF files and plots will be generated for complete end-to-end testing of the data processing path before the actual turn. When the actual deployment (the turn) happens we just need to change the name of the top-level deployment, e.g. from "Test M1 - November 2007" to "M1 - November 2007".

A. Procedure for setting up a test mooring deployment

This procedure coincides with standard operation procedure for assembling instruments on a mooring that is about to be deployed. As soon as the mooring configuration spreadsheet is distributed, the OASIS can is closed, and data begins flowing via the getM1 & getM2 scripts on tsunami these steps can be followed to set up a set of test deployments and corresponding netCDF files and plots using the DStoNetCDF.pl and combineTS.pl, combineAll.pl scripts in the DPforSSDS/cimt project. This test can help validate instrument metadata xml and the ssds.cfg file as well as produce a complete end-to-end test of the full SSDS data path.

  1. Copy SSDS instrument metadata config files from previous deployment to new deployment to create a starting place (these steps are best performed logged in as ssdsadmin on elvis), e.g.: 
    cd /mbari/ssdsdata/mooring/m1
cp -r 2006 2007

    (Of course you will copy the previous year's deployment configuration to a directory for the current year, hereafter referred to as <YYYY> or yyyy.) The new directory will have subdirectories named data, cfg, and xml. OSG will email a spreadsheet for the new deplolyment configuration; you may save it in the yyyy directory.

  2. Edit the mooring .cfg file and change the instrument deviceIDs and the path to the xml files to the newly deployed deviceIDs (aka ISI_IDs) and new xml directory. Note that the deviceID is repeated on each line: once in a field by itself and again in the name of the XML file describing the device deployment: 
    cd <YYYY>/cfg
vi ssds.cfg

    Refer to Bob's documentation for details on the formatting of the .cfg file. Generally, this is a tedious editing process where the deviceIDs from the OSG spreadsheet are copied into the proper field and into the name of the xml file that contains the device metadata.

  3. Get the most recent XML file for each instrument ID in the ssds.cfg file from the CVS puckxml module. Make sure that no deployment specific information is in the .xml files except for nominalDepth where appropriate. Another exception is the parent platform deployment XML file; its Deployment element should contain name, startDate, nominalLatitude and nominalLongitude attributes so that the parent deployment may be found using the Explorer application,  e.g.: 
    <Deployment role="platform" name="Test M1 - September 2008" startDate="2008-09-05T18:00:00Z" \
nominalLatitude="36.764" nominalLongitude="-122.046">

    In the XML make sure RecordVariable names are not set to standard coordinate axis names (longitude, latitude, depth, time), these are reserved for the OceanSITES data sets which derive from the insturment netCDF files produced with this metadata. Instead choose specific names, e.g. 'MetsysTime' for the Metsys time field. After cleaning up the XML check it back into CVS. Using the OASIS-specific schema, http://dods.mbari.org/data/ssdsdata/config/schema/2004oasis/SSDS_Metadata.xsd, will help with editing using a tool such as Oxygen or JEdit. Copy the files into the yyyy/xml subdirectory. Be careful with Device attributes, generally if a Device exists in the SSDS Device table with accurate attributes only the id needs to be specified in the xml file; if you include other attributes they will overwrite what is in the database.

  4. Special treatment for the TString "instrument": The inductive modem microcats are configured as child deployments of the Inductive modem Comm Device with one output that contains all the microcat data. Examine this xml file with a fine-tooth comb and make sure that the pressure sensor Record Variables are correct as well as the parseRegEx and nominalDepth attributes.
    [Note: With the M1 - October 2009 deployment we added the individual inductive modem microcats as children of the mooring and also configured a parallel deployment of the TString that produced the same data. With the M2 - April 2010 deployment we configured the individual IM mirocrocats and did not configure a parallel TString deployment. This is a better model for the system and is much easier to configure and produces more easily consumed data by downstream processes such as combineTS.pl.] With future deployments we will not configure TString.
  5. Make sure that the m1.cfg file in the cfg directory on oasis has a line referring to the correct ssds.cfg file, e.g.: 
    # The following is for ingesting OASIS data into SSDS
# Please leave it alone; 'extract' et al will ignore it
# 30 Oct 2007 Mike McCann
ssds	/ssdsdata/mooring/m1/2007/cfg/ssds.cfg
  6. Make sure that the ___ProcessRun lines in ssds.cfg refer to actual files in the xml directory.
  7. Check the ___ProcessRun.xml files for correct DataFile and Resource uri/urls.
  8. Make sure that the getM? script on tsunami executes the oasisToSSDS program on elvis.  The script should have lines like these: 
    set SEND_SSDS $TRUE

# Send the data to SSDS
# Added 20 Jan 2005 Bob Herlien
#
if ( $SEND_SSDS == $TRUE ) then
    echo "remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE"
    remsh elvis -l ssdsadmin dev/DPforSSDS/oasis/bin/oasisToSSDS -c $CFGFILE $RAW/$DATAFILE &
endif
  9. Before initiating the oasisToSSDS program take a printout of the ssds.cfg file over to the Mooring and verify the Device IDs for each instrument attached to the mooring. It's important to have this important metadata correct from the beginning otherwise the DataStreams will end up in the wrong "buckets" requiring some detailed re-work after the fact, namely following the instrument swap procedure and doing a deep delete (e.g.: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?responseType=text&delimiter=\|&objectToInvokeOn=DataProducerAccess&method=deepDelete&p1Type=DataProducer&p1Value=DataProducer|id=<deploymentID>) on the wrong instrument deployment.
  10. Test by:
    1. Monitor the extractRawData.log file in the ssdsdata/mooring/logs file for progress and any INFO or ERROR messages.  For instance if you see something like "[SSDS:main] INFO oasis.ssds.ingest.OasisToSSDS  - Unprocessed record, type = ASIMET_HRH" then that is an instrument missing from the ssds.cfg file for which data records are being received. To fix this add the correct instrument record to ssds.cfg.
    2. Check for metadata ingest in Explorer by searching for 'Test' in the "Search By Deployment Name:" field (note: may need to hit 'Enter' a few times to actually get something returned).  You should see a tree structure headed by the Deployment name entered in step 3. Open the deployment and drill down through the Attached Devices and Outputs to spot check the mooring instruments and their attributes - confirm against the the list provided by OSG. If there are any problems check the insturment XML and the ssds.cfg files configured in steps 2 to 4.
    3. To correct missing or wrong Deplyment attributes (e.g. nominalLatitude and nominalLongitude) the quickest way to correct is to use Enterprise Manager to select the DataProducer (identified by id in Explorer) and edt the fields directly in the table.
  11. Set up netCDF creation and plotting:
    1. Examine the processCurrentMooringDataStreams.sh script in /u/ssdsadmin/dev/DPforSSDS/cimt for lines (probably commented out) that process the test deployment.  They will look something like: 
      #
# Test M1 - Combine scripts look in Deployment Directory m1/200710
#
##set DDIR=200710
##DStoNetCDF.pl -mooring M1Test -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org \
-outputDir $ODIR -verbose -current |& tee $ODIR/m1/processing.log

      set DDIR to the YYYYMM of the test deployment and execute the DStoNetCDF.pl script with the above arguments; ODIR = /mbari/ssdsdata/deployments on elvis.  Note that the name of the Deployment is "Test M1 - <startDate>" in SSDS, but that the argument to the -mooring parameter is "M1Test". The DStoNetCDF.pl script contains a lookup that matches "M1Test" to "Test M1 ...".

    2. Execution of DstoNetCDF.pl will produce a web page in ssdsdata/deployments/m1test/current_qcPlots.html that contains links to the produced netCDF files and plots (available as web page http://dods.mbari.org/data/ssdsdata/deployments/m1test/current_qcPlots.html). Examine this page to see that every instrument we can process has a netCDF file and corresponding plots. If there are problems follow the instructions in the TROUBLESHOOT file in /u/ssdsadmin/dev/DPforSSDS/cimt. Generally, if the metadata is correct in SSDS then the netCDFs and plots will be produced. Either update the .cfg and XML files following steps 2-4 or update SSDS directly as appropriate.
    3. Configure the DStoNetCDF.pl to run for the test deployment in the cron job and inform OSG and the OASIS discussion list of the availability of these products.

B. Procedure for doing the actual production mooring turn

Close existing mooring deployment

  1. So, after the test has been run and data validated there is a sequence of manual steps for closing the previous deployment and beginning the production data processing of the new deployment. On the day the previous production mooring deployment is ended close that deployment by setting the endDate for the parent platform and child deployments.  From SSDS Explorer find the DataProducer ID for the mooring deployment, do a SELECT for that record and edit the endDate field (the times in the database are GMT). Then select all child deployments with a query on the foreign key like this:
SELECT     *
FROM         ssdsdba.DataProducer
WHERE     (ParentID_FK = 27122)

(Make sure to use the DataProducer ID for the platform deployment.) Set all endDates that are <NULL> to the actual end date.

For instruments that have child Sensor deployments (e.g. the Hyperspectral radiometers and imctd) do the same thing by changing the ParentID_FK to the id of the instrument and set all the <NULL> endDates so that everything on the recovered mooring is closed (or...as below -rschramm 4/2010) 

declare @myID as bigint
set @myID = 31548
SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK
FROM [ssdsdba].[DataProducer]
WHERE id = @myID and endDate is NULL
UNION
SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK
FROM [ssdsdba].[DataProducer]
WHERE parentid_fk = @myID  and endDate is NULL
UNION
SELECT id,name,dataProducerType,startDate,endDate,ParentID_FK
FROM [ssdsdba].[DataProducer]
WHERE parentid_fk in ( SELECT id
     FROM [ssdsdba].[DataProducer]
     WHERE parentid_fk = @myID  and endDate is NULL)
ORDER BY id

Configure new mooring deployment

  1. To control the new SSDS Metadata ingest temporarily turn off the oasisToSSDS execution in the getM? script on tsunami. This way you may edit the xml files at leisure without the ingest picking up any incorrect metadata while you are in the process of editing. In the XML file for the new platform deployment, which is currently configured as the Test deployment, edit the name, startDate, and nominalLatitude and nominalLongitude attributes to reflect the production deployment. E.g.: 
     <Deployment role="platform" name="M1 - October 2008" startDate="2008-10-08T17:00:00Z" nominalLatitude="36.764" nominalLongitude="-122.046">
		<Device id="1306"/>
		<Resource
			url="doc://foobar/watchCircle?centerLon=-122.0323&amp;centerLat=36.7562&amp;warningDist=1.2"
			name="Watch circle information">
			<description>Data for this mooring's watch circle are embedded in the uriString of this
				Resource. An application may parse for these parameters to use as criteria for
				issuing a warning if the GPS position is geater than warningDist (in km) from
				centerLat and centerLon (in decimal degrees WGS84)</description>
		</Resource>
	</Deployment>

    The nominalLatitude and nominalLongitude values should be exactly the same as all other deployments at M1 or M2. Make sure that the watchCircle parameters are relatively correct, the centerLon and centerLat values may be changed to reflect the actual anchor location. And these values can be updated as data come and in and we get a better idea of the actual watch circle. Save the changes to this file, check those changes into the puckxml CVS project and touch the remaining xml files so that SSDS ingest will recognize them as new.

  2. Do not close (set endDates) the existing Test deployment until after the oasisToSSDS has run with the new platform deployment name and all new instrument deployments have been created in SSDS_Metadata.
  3. See that oasisToSSDS is allowed to execute in the getM? script.  Monitor the /mbari/ssdsdata/mooring/logs/extractRawData.log file to see that downloaded records for the mooring being turned are processed. Then check that metadata is properly loaded with a query looking at the recently ingested Deployment metadata, e.g.: 
    SELECT     id, name, startDate, endDate, ParentID_FK
FROM       ssdsdba.DataProducer
WHERE      (dataProducerType = 'Deployment')
ORDER BY id DESC

    Things to check for:

    1. All ParentIDs should not be null except for the Platform deployment.
    2. Sensors should be children of instruments, e.g. inductive modem microcats are children of the Surface Inductive Modem
    3. All other instruments should be children of the platform deployment
  4. Check the SSDS_Metadata database for successful ingest of the platform and instrument deployment metadata with a query like this: 
    SELECT     ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name,
                      ssdsdba.Device.id AS DeviceID,
                      ssdsdba.Device.name AS DeviceName
FROM         ssdsdba.DataProducer INNER JOIN
                      ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id
WHERE     (ssdsdba.DataProducer.id  >= 28873)
ORDER BY ssdsdba.DataProducer.id DESC

    You will need to set the DataProducer.id criteria to be appropriate for this deployment.

  5. Add the new deployment to the DataProducerGroup "Mooring Deployments".  Do this query: 
    SELECT     ssdsdba.DataProducerAssocDataProducerGroup.*
FROM         ssdsdba.DataProducerAssocDataProducerGroup
WHERE     (DataProducerGroupID_FK = 111)

    and add a record to the DataProducerAssocDataProducerGroup table for the new platform deployment and the group id 111, which is the MooringDeployments group.

  6. You next need to link up the new deployment to a DataProducerGroup that contains all the deployment of the like designator. For example, if an M1 turn is being done, you need to add the new deployment to the "M1 Deployments" DataProducerGroup. You first need to find the DataProducerGroup.id for the group you are looking for. Do this query in Enterprise Manager: 
    SELECT * FROM ssdsdba.DataProducerGroup WHERE name = 'M1 Deployments'

    Make sure you use the correct M Number for the turn you are doing. Note the id from the results. Now you need to add the new deployment to that data producer group. Right-click on the DataProducerAssocDataProducerGroup table and do 'Open Table->Return All Rows'. You can then scroll to the bottom and there should be an open row that you can insert new values in. Put the ID of the new deployment in the first column and the id from the previous query in the second and then click out of the new row (that should commit the row). Do the same for the 'Mooring Deployments' DataProducerGroup.

  7. If a device is missing as a child of the mooring deployment then use the Raw Data Access Page to query for metadata from the device. For instance if the GPS is not a a child (as can now be seen from using SSDS Explorer and the predefined 'Mooring Deployments' query) then enter its device id and '0' for record type. From Enterprise manager you can then see which deployment and parent it got assigned to, e.g.: 
    SELECT     ssdsdba.DataProducer.id, ssdsdba.DataProducer.ParentID_FK AS ParentID, ssdsdba.DataProducer.name,
                      ssdsdba.Device.id AS DeviceID,
                      ssdsdba.Device.name AS DeviceName
FROM         ssdsdba.DataProducer INNER JOIN
                      ssdsdba.Device ON ssdsdba.DataProducer.DeviceID_FK = ssdsdba.Device.id
WHERE     (ssdsdba.Device.id = 1468)
ORDER BY ssdsdba.DataProducer.id DESC

    If there is no deployment for the time of the XML ingest then there was probably a deadlock on SSDS's attempt to insert it into the database. You can manually force an insert by altering the text of the device's xml file in moorings/m?/yyyy/xml (just add a space or something) and then run oasisToSSDS again, e.g.: > dev/DPforSSDS/oasis/bin/oasisToSSDS -c /oasis/cfg/m1.cfg /oasis/raw/m1.2007310.02, or wait for it to run with the hourly download on tsunami.

  8. Set up netCDF creation and plotting: In the processCurrentMooringDataStreams.sh script change the DDIR environment variable to the current YYYYMM for the production mooring run.  Comment out the Test deployment DStoNetCDF.pl execution and add lines for the closed deployment to the DEPLOYMENTS file.
  9. You will also need to edit the DStoNetCDF.pl script to set the new name of the M1 deployment, e.g.: 
    #
# SSDS mooring name Deployment lookup
#
my %ssdsMooringDeplNames = (
        M0 => 'M0',
        M1 => 'M1 - October 2009',
        M1Test => 'Test M1 - October 2009',
        M2 => 'M2 - April 2009',
  10. Monitor the SSDS processing web page (e.g. http://dods.mbari.org/data/ssdsdata/deployments/m1/current_qcPlots.html) for successful data processing.
  11. When the mooring log message is sent to oasis with the actual time of deployment enter that as the startDate in the database.  All times (DTGs) in the SSDS_Metadata database are GMT. Make sure to set the startDate for the imctd microcat sensors too. This can be done with a query like this where you use the proper dataProduceIDs for the mooring and the imcd: 
    SELECT     id, name, ParentID_FK, startDate, endDate, nominalDepth
FROM         ssdsdba.DataProducer
WHERE     (dataProducerType = 'Deployment') AND (ParentID_FK = 31564) OR
                      (ParentID_FK = 31548)
ORDER BY id DESC
  12. Because of some bug in SSDS ingest the dataContainerTypes of the outputs from the instrument deployments do not get properly assigned the values of 'Stream'.  This needs to be fixed so that the NDBC datatransfers will work. To fix it edit the SSDS_Metadata database starting with a query like this: 
    SELECT     ssdsdba.DataContainer.*
FROM         ssdsdba.DataProducer INNER JOIN
                      ssdsdba.DataContainer ON ssdsdba.DataProducer.id = ssdsdba.DataContainer.DataProducerID_FK
WHERE     (ssdsdba.DataProducer.ParentID_FK = 31548)

    where you use the DataProducerID for the new mooring deployment in the WHERE clause. ==> Change all of the 'File's in the dataContainerType' field to 'Stream's.

C. Procedures to be done after the mooring turn

Set up download info deployment and reporting

Though not a real instrument, we configure a virtual 'dlinfo' instrument for the download scripts to attach download statistics data. We re-use the same device IDs for the M1 and M2 moorings, so it's best to configure this after the new mooring is out and all of those deployments have been closed. As the data are delivered "out of band" from the typical OASIS instruments we need to create a deployment for dlinfo instrument by hand.

  1. The most direct way is to use the createDuplicateDeepDeployment service call. For example, to duplicate the 2009 M2 dlinfo deployment for the 2010 M2 deployment this call was executed: http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer\|id=34496&p2Type=Date&p2Value=2010-04-03T22:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&p4Value=2010-04-03T23:00:00Z&p5Type=String&p5Value=getM2-download&p6Type=String&p6Value=&delimiter=|. Of course you will need to adjust the DataProducer ID Date values for the new deployment you are creating. Here is the Key to the parameters:
    Key:
----
http://localhost:8080/servlet/MetadataAccessServlet
?responseType=text
&delimiter=|
&objectToInvokeOn=DataProducerAccess
&method=createDuplicateDeepDeployment
&p1Type=DataProducer
&p1Value=DataProducer|id=XXXX (XXXX is the ID of the deployment to copy)
&p2Type=Date
&p2Value=XXXXXX (XXXXXX is the start date of the new copy in XML format YYYY-MM-DDTHH:MM:SSZ)
&p3Type=boolean
&p3Value=(true|false)  (this is to indicate if you want the original deployment to be closed)
&p4Type=Date
&p4Value=XXXXXX (XXXXXX is the end date for the original deployment (if p3Value is true))
&p5Type=String
&p5Value=XXXXXX (XXXXXX is the DataProducer name of the new DataProducer)
&p6Type=String
&p6Value=XXXXXX (XXXXXX is the base URL to use for the new DataContainers that will be created).
&delimiter=|

    Executing the createDuplicateDeepDeployment service call will return an ID for the new deployment.
  1. Edit the new deployment record in the DataProducer table to adjust it's parentID_FK to be for the new mooring. While there edit times and name as appropriate.
  2. Edit the getM? script on tsunami to use the proper device and parent ID (this will be the device ID of the torroid of the mooring deployment) and make sure that the '/oasis/bin/ssdsSubmit.pl $deviceId $parentId "$starttime_es,$endtime_es,$filesize,$rtnsts"' line in the script is configured to run.

Here's another example for the October 2010 M1 turn:

http://new-ssds.mbari.org:8080/servlet/MetadataAccessServlet?method=createDuplicateDeepDeployment&objectToInvokeOn=DataProducerAccess&\
p1Type=moos.ssds.metadata.DataProducer&p1Value=DataProducer|id=35591&p2Type=Date&p2Value=2010-10-27T21:00:00Z&p3Type=boolean&p3Value=false&p4Type=Date&\
p4Value=2010-q0-27T20:30:00Z&p5Type=String&p5Value=getM1-download&p6Type=String&p6Value=&delimiter=|

(This returned a new DataProducerID of 41556.  The ParentID_FK for this record in the DataProducer table was then changed from 35388 to 
41197 and the endDate cleared to null.
Also, with the change to alternating deviceIDs implemented in 2010 the deviceID_FK was changed from 1698 to 1768 for this M1 deployment.

N.B. A rotating scheme for the virtual device IDs was implemented in 2010. This permits the steps described above to be executed prior to the mooring turn during the extensive
dock-side test period.

Enable processing for other "virtual" devices

  1. Turn on ClockSync processing. We also use a virtual device ID for these data. Simply uncomment the line for it in the ssds.cfg file.
  2. Turn on ISUS processing. Simply uncomment the line for it in the ssds.cfg file. (Device ID re-used from previous deployment.)

Make all of the child instrument deployment start times the same as the mooring start time

When SSDS receives a packet from an instrument that is not currently deployed it will create a Deployment record (in the DataProducer table) with a startDate that is set to the time of the first record received. As the mooring starts up all of the instrument deployments will have different start times based on when they each first saw data. This can present problems for the data processing that follows, especially for the jobs that aggregate the microcat data into a single ZT file that is used to produce the contour temperature and salinity wind stick plots. To prevent these problems it's best to edit the startDates of the child instrument deployments so that they are all the same. This is currently most easily done through Enterprise Manager with a query like below (this is for the 201010 M1 deployment) to get all the child instrument deployments:

SELECT    *
FROM      ssdsdba.DataProducer
WHERE     (ParentID_FK = 41197)
ORDER BY  startDate

Then copy and paste the datetime string from one cell to the next. The times are GMT.

Cycle links to previous deployment

  1. Edit previous.html file in /mbari/ssdsdata/deployments to add a line for the new deployment and add the end date and archive url for the just closed deployment, e.g.:
    <li><a href="http://dods.mbari.org/data/ssdsdata/deployments/m2/m2_200904_qcPlots.html"  >200904</a> 2009-04-29 to 2010-04-03</li>
<li><a href="http://dods.mbari.org/data/ssdsdata/deployments/m2/current_qcPlots.html"  >201004</a> 2010-04-03 to present</li>

  2. Add lines to DEPLOYMENTS file in dev/DPforSSDS/cimt/ for the just closed deployment, e.g.:
      DStoNetCDF.pl -mooring M2 -deployment "M2 - April 2009" -ssdsServer new-ssds.mbari.org -ssdsDataServer new-ssds.mbari.org -outputDir \
/mbari/ssdsdata/deployments -procClosed -verbose
  combineM.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments
  combineTS.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments
  combineAll.pl -mooring M2 -ssdsServer new-ssds.mbari.org -deployments 200904 -ssdsDataServer new-ssds.mbari.org -inputDir /mbari/ssdsdata/deployments

and

  /bin/cp /mbari/ssdsdata/deployments/m2/200904/OS_*.nc /mbari/FTP/pub/OceanSITES

  3. Execute these lines to generate "closed deployment" data products and web pages and for submission to the OceanSITES GDAC
  4. Confirm that the processing executed properly. Sometimes mangled timestamp data weasels its way into the data stream causing instrument netcdf files to be named with bogus start dates. (This can also happen during a deployment and is one of the maintenance tasks one should follow to keep the data flowing to where it needs to go.) The combine__.pl scripts will then create the OceanSITES formatted files encompassing the dates of all the instrument netcdf files. Incorrect instrument file names will create incorrect OS_MBARI* file names causing CenCOOS and NDBC/Ifremer to complain about the correctly named file not being updated with new data. The fix involves a purging of the bad files, double checking the metadata in SSDS and reprocessing, with perhaps additional checks in the instrument processing perl code to skip over bad records. This is best done at the Unix command line by cd'ing to the deployment directory, e.g. 
    cd /mbari/deployments/m1/201010
ls -lrt

    and removing files that have bogus dates that do not represent the deployment start date. You should also remove all the directories and files in the gifs/ subdirectory. These will all get recreated when you run DStoNetCDF.pl and the combine__.pl scripts. With luck, simply removing the bad files and rerunning the processing scripts will fix things. If not, identify where the bad dates are coming from and fix as appropriate.

Mike McCann (First edit: 30 October 2007, Last updated: 1 March 2012)

Comments:

On the day of the turn...

the place you change the "top-level deployment name" is on elvis as ssdsadmin. In the /hosts/tornado_vol0/ssdsdata/mooring/m?/200?/xml/nnnn.xml where nnnn.xml is the device id for the oasis buoy itself (the fiberglass doughnut....). Also looks like you set the startDate there as well. Not sure if the startDate can be set later or does it have to be correct from the get-go...
Posted by at Apr 11, 2008 09:37

Also, Kevin indicates it is necessary to 'touch' all of the xml files in /hosts/tornado_vol0/ssdsdata/mooring/m?/200?/xml/

as part of the turn process to make the new xml ingest.
Posted by at Apr 11, 2008 09:40

The critical missing piece is documented at:

http://oceana.shore.mbari.org:8082/browse/CD-10
Posted by at Apr 11, 2008 10:09
Document generated by Confluence on Feb 04, 2026 08:35